{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prompt for Agent\n",
    "\n",
    "- Author: [Secludor](https://github.com/Secludor)\n",
    "- Peer Review: \n",
    "- This is a part of [LangChain Open Tutorial](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial)\n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/99-TEMPLATE/00-BASE-TEMPLATE-EXAMPLE.ipynb) [![Open in GitHub](https://img.shields.io/badge/Open%20in%20GitHub-181717?style=flat-square&logo=github&logoColor=white)](https://github.com/LangChain-OpenTutorial/LangChain-OpenTutorial/blob/main/99-TEMPLATE/00-BASE-TEMPLATE-EXAMPLE.ipynb)\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook provides practical examples for building and evaluating dynamic AI agent prompts using LangChain. The notebook is organized around two main templates:\n",
    "\n",
    "1. **AI_CAREER_PLANNER_AGENT_PROMPT**  \n",
    "   This section demonstrates how to construct a robust prompt template that generates personalized study plans for users aspiring to become AI engineers. It covers:\n",
    "   - **Tool Definitions**: Functions such as ```resource_recommender``` and ```timeline_generator``` are implemented to tailor learning resources and study timelines according to the user's skill gaps.\n",
    "   - **Prompt Template**: The prompt template clearly defines the agent's role and behavioral guidelines, establishes working memory management, and integrates tools for generating actionable, structured study plans.\n",
    "   - **Memory Initialization and Response Formatting**: It ensures that conversation history is maintained and outputs are formatted in a clear Markdown structure.\n",
    "   - **Runnable Chain**: All components are integrated into a chain that processes user input, invokes the tools, and produces a final study plan, demonstrating flexibility and user-specific customization.\n",
    "\n",
    "2. **PROMPT_QUALITY_EVALUATION_TEMPLATE**  \n",
    "   This section illustrates a two-part process:\n",
    "   - **Prompt Generation**: A generation template is employed to create prompts based on a user’s goal (e.g., developing a study plan for an AI engineering career).\n",
    "   - **Prompt Evaluation**: An evaluation template is defined for a Prompt Quality Reviewer agent that verifies the generated prompt against defined criteria such as identity clarity, memory management, tool integration, plan completeness, and overall quality.  \n",
    "     *Note*: Every literal ```{``` and ```}``` inside the JSON block is replaced with ```{{``` and ```}}``` to avoid unintended substitutions during Python’s formatting process.\n",
    "\n",
    "Together, these templates showcase a modular and scalable approach to designing, generating, and evaluating dynamic prompts for AI agents. By integrating tool functions, context-aware memory management, and rigorous evaluation criteria, the notebook provides concrete examples of how to achieve intelligent task processing and high-quality agent output in real-world scenarios.\n",
    "\n",
    "### Table of Contents\n",
    "- [Overview](#overview)\n",
    "- [Environment Setup](#environment-setup)\n",
    "- [PROMPT_QUALITY_EVALUATION_TEMPLATE](#prompt_quality_evaluation_template)\n",
    "- [AI_CAREER_PLANNER_AGENT_PROMPT](#ai_career_planner_agent_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Setup\n",
    "\n",
    "Set up the environment. You may refer to [Environment Setup](https://wikidocs.net/257836) for more details.\n",
    "\n",
    "**[Note]**\n",
    "- ```langchain-opentutorial``` is a package that provides a set of easy-to-use environment setup, useful functions and utilities for tutorials. \n",
    "- You can checkout the [```langchain-opentutorial```](https://github.com/LangChain-OpenTutorial/langchain-opentutorial-pypi) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 24.3.1 -> 25.0.1\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install -U langchain-opentutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install required packages\n",
    "from langchain_opentutorial import package\n",
    "\n",
    "package.install(\n",
    "    [\n",
    "        \"langsmith\",\n",
    "        \"langchain_core\",\n",
    "        \"langchain_community\",\n",
    "        \"langchain_openai\",\n",
    "    ],\n",
    "    verbose=False,\n",
    "    upgrade=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can set API keys in a ```.env``` file or set them manually.\n",
    "\n",
    "[Note] If you’re not using the ```.env``` file, no worries! Just enter the keys directly in the cell below, and you’re good to go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Environment variables have been set successfully.\n"
     ]
    }
   ],
   "source": [
    "from dotenv import load_dotenv\n",
    "from langchain_opentutorial import set_env\n",
    "\n",
    "# Attempt to load environment variables from a .env file; if unsuccessful, set them manually.\n",
    "if not load_dotenv():\n",
    "    set_env(\n",
    "        {\n",
    "            \"OPENAI_API_KEY\": \"\",\n",
    "            \"LANGCHAIN_API_KEY\": \"\",\n",
    "            \"LANGCHAIN_TRACING_V2\": \"true\",\n",
    "            \"LANGCHAIN_ENDPOINT\": \"https://api.smith.langchain.com\",\n",
    "            \"TAVILY_API_KEY\": \"\",\n",
    "        }\n",
    "    )\n",
    "\n",
    "# set the project name same as the title\n",
    "set_env(\n",
    "    {\n",
    "        \"LANGCHAIN_PROJECT\": \"agentic-prompt\",\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AI_CAREER_PLANNER_AGENT_PROMPT\n",
    "\n",
    "This section demonstrates the implementation of the **AI_CAREER_PLANNER_AGENT_PROMPT**, which is designed to generate personalized study plans for users aiming to become AI engineers. Below is a detailed explanation of each part of the code:\n",
    "\n",
    "### **1. Tool Definitions**\n",
    "- **Purpose**: Two tools are defined to assist in creating the study plan:\n",
    "  - **```resource_recommender```**: Recommends learning resources based on the user's current skills and target skills.\n",
    "    - Input: User's current skills and desired target skills.\n",
    "    - Output: A list of recommended resources, including course names, descriptions, and URLs.\n",
    "  - **```timeline_generator```**: Generates a milestone-based study timeline.\n",
    "    - Input: User's current skills and available study time (in weeks).\n",
    "    - Output: A structured timeline with weekly goals.\n",
    "\n",
    "- **Key Features**:\n",
    "  - The tools ensure that the recommendations and timelines are tailored to the user's skill gaps.\n",
    "  - Default target skills include Python, Linear Algebra, and Deep Learning, which are essential for AI engineering.\n",
    "\n",
    "### **2. Prompt Template**\n",
    "- **Purpose**: The ```ChatPromptTemplate``` defines the behavior and role of the AI agent. It provides clear instructions for generating a structured study plan.\n",
    "- **Key Elements**:\n",
    "  - **Agent Identity**: Specifies that the agent is an \"AI Career Planning Specialist\" with expertise in AI research and education.\n",
    "  - **Behavioral Guidelines**:\n",
    "    - Analyze skill gaps.\n",
    "    - Recommend accessible learning resources.\n",
    "    - Create realistic timelines with milestones.\n",
    "  - **Working Memory Management**:\n",
    "    - Tracks the current task context, progress, and pending actions.\n",
    "    - Maintains awareness of conversation history and user input.\n",
    "  - **Tool Integration**:\n",
    "    - Utilizes ```resource_recommender``` for learning resources.\n",
    "    - Uses ```timeline_generator``` for creating timelines.\n",
    "\n",
    "- **Output Requirements**:\n",
    "  - The response must be in Markdown format with sections for skill gap analysis, recommended resources, a timeline, and progress tracking checkpoints.\n",
    "\n",
    "### **3. Memory Initialization**\n",
    "- **Purpose**: Initializes memory using ```ConversationKGMemory``` to maintain conversation history and context across interactions.\n",
    "- **Error Handling**:\n",
    "  - If memory initialization fails, it falls back to a default empty memory state to ensure robustness.\n",
    "\n",
    "### **4. Response Formatting**\n",
    "- **Purpose**: Formats the output from the LLM or tool chain into a structured Markdown response for better readability.\n",
    "- **Key Features**:\n",
    "  - Combines recommendations and timeline into a single cohesive response.\n",
    "  - Ensures that each section (resources, timeline) is clearly presented with actionable details.\n",
    "\n",
    "### **5. Runnable Chain**\n",
    "- **Purpose**: Combines all components into a runnable chain that processes user input and generates a personalized study plan.\n",
    "- **Steps in the Chain**:\n",
    "  1. Passes user input directly using ```RunnablePassthrough```.\n",
    "  2. Loads conversation history using ```RunnableLambda(load_memory)```.\n",
    "  3. Executes tools (```resource_recommender```, ```timeline_generator```) to generate recommendations and timelines.\n",
    "  4. Processes inputs through the prompt template (```prompt```) to guide the LLM's behavior.\n",
    "  5. Formats the final output using ```RunnableLambda(format_response)```.\n",
    "\n",
    "### **6. Example Usage**\n",
    "- Demonstrates how to use the runnable chain to generate a study plan for a user aiming to become an AI engineer.\n",
    "- Example Input:\n",
    "  ```python\n",
    "  {\n",
    "      \"input\": \"I want to become an AI engineer. I know {current_skills}.\",\n",
    "      \"current_skills\": [\"Basic Python\"],\n",
    "      \"target_skills\": [\"Python\", \"Linear Algebra\", \"Deep Learning\"],\n",
    "      \"weeks_available\": 12,\n",
    "  }\n",
    "  ```\n",
    "- Example Output (Formatted Markdown):\n",
    "  ```markdown\n",
    "  # AI Engineering Career Study Plan\n",
    "\n",
    "  ## Recommended Resources\n",
    "  - **Python**: [Python for Everybody (Coursera)](https://www.coursera.org/specializations/python)\n",
    "  - **Linear Algebra**: [Khan Academy Linear Algebra](https://www.khanacademy.org/math/linear-algebra)\n",
    "  - **Deep Learning**: [Deep Learning Specialization](https://www.coursera.org/specializations/deep-learning)\n",
    "\n",
    "  ## Study Timeline\n",
    "  - Week 1: Python Basics (2 weeks)\n",
    "  - Week 3: Linear Algebra Fundamentals (2 weeks)\n",
    "  - Week 5: Deep Learning Foundations (4 weeks)\n",
    "  - Week 9: Practical Projects (4 weeks)\n",
    "\n",
    "  ## Progress Tracking\n",
    "  Weekly self-assessments and milestone reviews are recommended to ensure steady progress.\n",
    "  ```\n",
    "\n",
    "### Summary\n",
    "The **AI_CAREER_PLANNER_AGENT_PROMPT** section showcases how to integrate tools, memory management, and prompt templates into a cohesive system for generating actionable study plans. By leveraging LangChain's modular components, this implementation ensures flexibility, scalability, and user-specific customization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# AI Career Preparation Plan\n",
      "\n",
      "## Overview\n",
      "This plan is designed to help you transition from your current skill level to becoming an AI engineer. Based on your current skills and target skills, we will focus on enhancing your Python proficiency, building a strong foundation in Linear Algebra, and introducing you to Deep Learning concepts.\n",
      "\n",
      "## Current Skills\n",
      "- Basic Python\n",
      "\n",
      "## Target Skills\n",
      "1. Intermediate to Advanced Python\n",
      "2. Linear Algebra\n",
      "3. Deep Learning\n",
      "\n",
      "## Study Timeline\n",
      "You have 12 weeks available for study. The timeline below outlines a structured approach to achieve your target skills within this period.\n",
      "\n",
      "### Weeks 1-4: Python Proficiency\n",
      "**Objective:** Transition from basic to intermediate/advanced Python skills.\n",
      "\n",
      "- **Resources:**\n",
      "  - [Automate the Boring Stuff with Python](https://automatetheboringstuff.com/) - Free online book and course\n",
      "  - [Python for Everybody](https://www.coursera.org/specializations/python) - Coursera specialization (audit for free)\n",
      "  - [LeetCode](https://leetcode.com/) - Practice Python coding problems\n",
      "\n",
      "- **Milestones:**\n",
      "  - Week 1: Complete \"Automate the Boring Stuff with Python\" chapters 1-6\n",
      "  - Week 2: Complete \"Automate the Boring Stuff with Python\" chapters 7-12\n",
      "  - Week 3: Start \"Python for Everybody\" and complete first two courses\n",
      "  - Week 4: Solve 10 Python problems on LeetCode\n",
      "\n",
      "### Weeks 5-8: Linear Algebra\n",
      "**Objective:** Develop a solid understanding of Linear Algebra, essential for AI and ML.\n",
      "\n",
      "- **Resources:**\n",
      "  - [Khan Academy Linear Algebra](https://www.khanacademy.org/math/linear-algebra) - Free online course\n",
      "  - [Essence of Linear Algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9Kmd6CLG0GF0jLOpcyb) - YouTube series by 3Blue1Brown\n",
      "\n",
      "- **Milestones:**\n",
      "  - Week 5: Complete Khan Academy's \"Vectors and Spaces\" section\n",
      "  - Week 6: Complete Khan Academy's \"Matrix Transformations\" section\n",
      "  - Week 7: Watch and understand 3Blue1Brown's videos 1-5\n",
      "  - Week 8: Watch and understand 3Blue1Brown's videos 6-10\n",
      "\n",
      "### Weeks 9-12: Deep Learning\n",
      "**Objective:** Gain introductory knowledge of Deep Learning concepts and frameworks.\n",
      "\n",
      "- **Resources:**\n",
      "  - [Deep Learning Specialization by Andrew Ng](https://www.coursera.org/specializations/deep-learning) - Coursera specialization (audit for free)\n",
      "  - [Fast.ai Practical Deep Learning for Coders](https://course.fast.ai/) - Free online course\n",
      "\n",
      "- **Milestones:**\n",
      "  - Week 9: Complete the first course of the Deep Learning Specialization\n",
      "  - Week 10: Complete the second course of the Deep Learning Specialization\n",
      "  - Week 11: Start Fast.ai course and complete the first two lessons\n",
      "  - Week 12: Complete Fast.ai lessons 3 and 4\n",
      "\n",
      "## Progress Tracking\n",
      "- **Checkpoints:**\n",
      "  - End of Week 4: Assess Python proficiency through coding challenges\n",
      "  - End of Week 8: Test understanding of Linear Algebra with practice problems\n",
      "  - End of Week 12: Implement a simple neural network using a framework like TensorFlow or PyTorch\n",
      "\n",
      "## Additional Tips\n",
      "- Allocate consistent daily study time to maintain momentum.\n",
      "- Join online forums or study groups for peer support and motivation.\n",
      "- Regularly review and adjust your study plan based on progress and understanding.\n",
      "\n",
      "By following this plan, you will build a strong foundation in the essential skills needed to pursue a career as an AI engineer. Good luck on your learning journey!\n"
     ]
    }
   ],
   "source": [
    "from typing import Dict, List\n",
    "from langchain_community.memory.kg import ConversationKGMemory\n",
    "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
    "from langchain_core.runnables import RunnableLambda, RunnablePassthrough\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "# Initialize the LLM with the desired model and configuration.\n",
    "llm = ChatOpenAI(model_name=\"gpt-4o\", temperature=0)\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Define Tools for Resource Recommendation and Timeline Generation\n",
    "# -----------------------------------------------------------\n",
    "\n",
    "\n",
    "def resource_recommender(input_data: Dict) -> Dict:\n",
    "    \"\"\"\n",
    "    Recommend learning resources based on the user's current skills\n",
    "    and target skills for an AI engineering career.\n",
    "\n",
    "    Input: {\n",
    "      \"current_skills\": List,\n",
    "      \"target_skills\": List (default: [\"Python\", \"Linear Algebra\", \"Deep Learning\"])\n",
    "    }\n",
    "    Returns a dictionary with a list of recommendations.\n",
    "    \"\"\"\n",
    "    current_skills = input_data.get(\"current_skills\", [])\n",
    "    target_skills = input_data.get(\n",
    "        \"target_skills\", [\"Python\", \"Linear Algebra\", \"Deep Learning\"]\n",
    "    )\n",
    "    recommendations = []\n",
    "\n",
    "    for skill in target_skills:\n",
    "        if skill not in current_skills:\n",
    "            if skill == \"Python\":\n",
    "                recommendations.append(\n",
    "                    {\n",
    "                        \"skill\": skill,\n",
    "                        \"resource\": \"Python for Everybody (Coursera)\",\n",
    "                        \"url\": \"https://www.coursera.org/specializations/python\",\n",
    "                    }\n",
    "                )\n",
    "            elif skill == \"Linear Algebra\":\n",
    "                recommendations.append(\n",
    "                    {\n",
    "                        \"skill\": skill,\n",
    "                        \"resource\": \"Khan Academy Linear Algebra\",\n",
    "                        \"url\": \"https://www.khanacademy.org/math/linear-algebra\",\n",
    "                    }\n",
    "                )\n",
    "            elif skill == \"Deep Learning\":\n",
    "                recommendations.append(\n",
    "                    {\n",
    "                        \"skill\": skill,\n",
    "                        \"resource\": \"Deep Learning Specialization\",\n",
    "                        \"url\": \"https://www.coursera.org/specializations/deep-learning\",\n",
    "                    }\n",
    "                )\n",
    "    return {\"recommendations\": recommendations}\n",
    "\n",
    "\n",
    "def timeline_generator(input_data: Dict) -> Dict:\n",
    "    \"\"\"\n",
    "    Generate a milestone-based study timeline based on user input.\n",
    "\n",
    "    Input: {\n",
    "      \"current_skills\": List,\n",
    "      \"weeks_available\": Integer (default: 12)\n",
    "    }\n",
    "    Returns a dictionary containing a timeline.\n",
    "    \"\"\"\n",
    "    current_skills = input_data.get(\"current_skills\", [])\n",
    "    weeks_available = input_data.get(\n",
    "        \"weeks_available\", 12\n",
    "    )  # It doesn't use in this example.\n",
    "    timeline = []\n",
    "    week = 1\n",
    "\n",
    "    # Schedule Python basics if not mastered.\n",
    "    if \"Python\" not in current_skills:\n",
    "        timeline.append({\"week\": week, \"goal\": \"Python Basics\", \"duration\": \"2 weeks\"})\n",
    "        week += 2\n",
    "\n",
    "    # Continue with other core subjects.\n",
    "    timeline.extend(\n",
    "        [\n",
    "            {\n",
    "                \"week\": week,\n",
    "                \"goal\": \"Linear Algebra Fundamentals\",\n",
    "                \"duration\": \"2 weeks\",\n",
    "            },\n",
    "            {\n",
    "                \"week\": week + 2,\n",
    "                \"goal\": \"Deep Learning Foundations\",\n",
    "                \"duration\": \"4 weeks\",\n",
    "            },\n",
    "            {\"week\": week + 6, \"goal\": \"Practical Projects\", \"duration\": \"4 weeks\"},\n",
    "        ]\n",
    "    )\n",
    "    return {\"timeline\": timeline}\n",
    "\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Define the Prompt Template for the AI Career Planner Agent.\n",
    "# -----------------------------------------------------------\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \"\"\"You are a specialized agent with the following identity and operational parameters:\n",
    "\n",
    "Identity:\n",
    "- Role: AI Career Planning Specialist\n",
    "- Domain: AI Research and Engineering Education\n",
    "- Purpose: Create personalized AI career preparation plans\n",
    "- Interaction Style: Structured, supportive, and detail-oriented\n",
    "\n",
    "Behavioral Guidelines:\n",
    "- Primary Goals:\n",
    "  - Analyze skill gaps in AI/ML knowledge\n",
    "  - Recommend targeted learning resources\n",
    "  - Generate achievable study timelines\n",
    "- Constraints:\n",
    "  - Use only verified and accessible learning resources\n",
    "  - Maintain realistic time expectations\n",
    "  - Consider user's current skill level\n",
    "- Success Criteria:\n",
    "  - Clear progression path with measurable milestones\n",
    "  - Resource recommendations matched to skill level\n",
    "  - Achievable timeline based on user availability\n",
    "\n",
    "Current Date: February 12, 2025\n",
    "\n",
    "Working Memory Management:\n",
    "- Current Task Context:\n",
    "  - Active objective: Create personalized AI career study plan\n",
    "  - Task progress: Analyzing user input and requirements\n",
    "  - Pending actions: Resource matching and timeline creation\n",
    "- Recent Interaction State:\n",
    "  - Last user input: {input}\n",
    "  - Conversation history: {history}\n",
    "  - Current focus: Skill gap analysis and planning\n",
    "\n",
    "Tool Integration:\n",
    "- Resource Recommender: Match learning resources to identified skill gaps\n",
    "- Timeline Generator: Create structured study schedule\n",
    "- Progress Tracker: Monitor and adjust study milestones\n",
    "\n",
    "Output Requirements:\n",
    "1. Structured Markdown format with clear sections\n",
    "2. Specific, actionable recommendations\n",
    "3. Timeline with realistic milestones\n",
    "4. Progress tracking checkpoints\n",
    "\n",
    "Please maintain awareness of:\n",
    "- User's current skill level\n",
    "- Available study time\n",
    "- Learning resource accessibility\n",
    "- Skill dependencies and prerequisites\n",
    "\"\"\",\n",
    "        ),\n",
    "        MessagesPlaceholder(variable_name=\"history\"),\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Initialize and Integrate Memory with Error Handling\n",
    "# -----------------------------------------------------------\n",
    "\n",
    "try:\n",
    "    memory = ConversationKGMemory(llm=llm)\n",
    "except Exception as e:\n",
    "    print(f\"Memory initialization error: {e}\")\n",
    "    memory = None\n",
    "\n",
    "\n",
    "def load_memory(input_dict: Dict) -> List:\n",
    "    \"\"\"\n",
    "    Load conversation history from memory as a list of messages,\n",
    "    ensuring the correct data format.\n",
    "    \"\"\"\n",
    "    if memory is None:\n",
    "        return []  # Return an empty history if memory is unavailable.\n",
    "    try:\n",
    "        history = memory.load_memory_variables(\n",
    "            {\"input\": input_dict.get(\"input\", \"\")}\n",
    "        ).get(\"history\", [])\n",
    "        if not isinstance(history, list):\n",
    "            history = []\n",
    "        return history\n",
    "    except Exception as e:\n",
    "        print(f\"Memory loading error: {e}\")\n",
    "        return []\n",
    "\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Response Formatting Function\n",
    "# -----------------------------------------------------------\n",
    "\n",
    "\n",
    "def format_response(llm_response) -> str:\n",
    "    \"\"\"\n",
    "    Format the LLM output (or a consolidated tool output dictionary)\n",
    "    into a structured Markdown plan.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        # If llm_response is an object with a 'content' attribute.\n",
    "        if hasattr(llm_response, \"content\"):\n",
    "            return llm_response.content\n",
    "        if isinstance(llm_response, str):\n",
    "            return llm_response\n",
    "        if isinstance(llm_response, dict):\n",
    "            recommendations = llm_response.get(\"recommendations\", [])\n",
    "            timeline = llm_response.get(\"timeline\", [])\n",
    "            response = \"# AI Engineering Career Study Plan\\n\\n\"\n",
    "            response += \"## Recommended Resources\\n\"\n",
    "            for rec in recommendations:\n",
    "                response += f\"- **{rec['skill']}**: [{rec['resource']}]({rec['url']})\\n\"\n",
    "            response += \"\\n## Study Timeline\\n\"\n",
    "            for item in timeline:\n",
    "                response += (\n",
    "                    f\"- Week {item['week']}: {item['goal']} ({item['duration']})\\n\"\n",
    "                )\n",
    "            return response\n",
    "        return str(llm_response)\n",
    "    except Exception as e:\n",
    "        return f\"Error formatting response: {str(e)}\\nRaw response: {str(llm_response)}\"\n",
    "\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Build the Runnable Chain for the Agent\n",
    "# -----------------------------------------------------------\n",
    "\n",
    "chain = (\n",
    "    {\n",
    "        \"input\": RunnablePassthrough(),  # Pass the user input unchanged.\n",
    "        \"history\": RunnableLambda(load_memory),  # Load conversation history.\n",
    "        \"recommendations\": RunnableLambda(\n",
    "            lambda x: resource_recommender(x)\n",
    "        ),  # Get resource recommendations.\n",
    "        \"timeline\": RunnableLambda(\n",
    "            lambda x: timeline_generator(x)\n",
    "        ),  # Generate study timeline.\n",
    "    }\n",
    "    | prompt  # Feed the consolidated inputs into the prompt.\n",
    "    | llm  # Generate a refined response using the LLM.\n",
    "    | RunnableLambda(format_response)  # Format the final output.\n",
    ")\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Example Usage: Generate a Study Plan for AI Research/Engineering Career Preparation\n",
    "# -----------------------------------------------------------\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    try:\n",
    "        # Example user input with dynamic parameters indicated with {}.\n",
    "        user_input = {\n",
    "            \"input\": \"I want to become an AI engineer. I know {current_skills}.\",\n",
    "            \"current_skills\": [\"Basic Python\"],\n",
    "            \"target_skills\": [\"Python\", \"Linear Algebra\", \"Deep Learning\"],\n",
    "            \"weeks_available\": 12,\n",
    "        }\n",
    "\n",
    "        # Execute the chain to produce a structured study plan.\n",
    "        response = chain.invoke(user_input)\n",
    "        print(response)\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"Error executing chain: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Personalized AI Career Study Plan\n",
      "\n",
      "## 1. Skill Gap Analysis\n",
      "\n",
      "Based on your current skills and target skills, here is the analysis:\n",
      "\n",
      "- **Current Skills**: Basic Python\n",
      "- **Target Skills**: \n",
      "  - Advanced Python\n",
      "  - Linear Algebra\n",
      "  - Deep Learning\n",
      "\n",
      "### Identified Skill Gaps:\n",
      "1. **Advanced Python**: You need to enhance your Python skills to include data manipulation, libraries like NumPy and Pandas, and object-oriented programming.\n",
      "2. **Linear Algebra**: Fundamental understanding of vectors, matrices, and operations is crucial for AI.\n",
      "3. **Deep Learning**: Understanding neural networks, backpropagation, and frameworks like TensorFlow or PyTorch.\n",
      "\n",
      "## 2. Recommended Resources\n",
      "\n",
      "Here are some verified and accessible resources to help you fill these gaps:\n",
      "\n",
      "### Advanced Python\n",
      "- **Course**: [Python for Data Science and AI](https://www.coursera.org/learn/python-for-applied-data-science-ai) - Coursera\n",
      "- **Book**: [Automate the Boring Stuff with Python](https://automatetheboringstuff.com/)\n",
      "\n",
      "### Linear Algebra\n",
      "- **Course**: [Linear Algebra for Beginners: Open Doors to Great Careers](https://www.udemy.com/course/linear-algebra-for-beginners-open-doors-to-great-careers/) - Udemy\n",
      "- **Book**: [Linear Algebra and Its Applications](https://www.amazon.com/Linear-Algebra-Its-Applications-5th/dp/032198238X) by David C. Lay\n",
      "\n",
      "### Deep Learning\n",
      "- **Course**: [Deep Learning Specialization](https://www.coursera.org/specializations/deep-learning) - Coursera\n",
      "- **Book**: [Deep Learning](https://www.deeplearningbook.org/) by Ian Goodfellow, Yoshua Bengio, and Aaron Courville\n",
      "\n",
      "## 3. Study Timeline with Milestones\n",
      "\n",
      "### Weeks 1-4: Advanced Python\n",
      "- **Week 1-2**: Complete \"Python for Data Science and AI\" course.\n",
      "- **Week 3**: Read and practice exercises from \"Automate the Boring Stuff with Python\".\n",
      "- **Week 4**: Work on a small project using Python libraries like NumPy and Pandas.\n",
      "\n",
      "### Weeks 5-7: Linear Algebra\n",
      "- **Week 5-6**: Complete \"Linear Algebra for Beginners\" course.\n",
      "- **Week 7**: Solve problems from \"Linear Algebra and Its Applications\" to solidify understanding.\n",
      "\n",
      "### Weeks 8-12: Deep Learning\n",
      "- **Week 8-10**: Complete the first three courses of the \"Deep Learning Specialization\".\n",
      "- **Week 11**: Read selected chapters from \"Deep Learning\" by Goodfellow et al.\n",
      "- **Week 12**: Implement a simple neural network using TensorFlow or PyTorch.\n",
      "\n",
      "## 4. Next Steps and Progress Tracking\n",
      "\n",
      "- **Weekly Review**: Set aside time each week to review what you've learned and adjust your study plan if necessary.\n",
      "- **Project Portfolio**: Start building a portfolio of projects to showcase your skills to potential employers.\n",
      "- **Community Engagement**: Join AI communities or forums to stay updated and seek help when needed.\n",
      "\n",
      "By following this structured plan, you will be well on your way to becoming an AI engineer. Remember to track your progress and adjust the plan as needed to fit your learning pace. Good luck!\n"
     ]
    }
   ],
   "source": [
    "from typing import Dict, List, Any\n",
    "from langchain_community.memory.kg import ConversationKGMemory\n",
    "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
    "from langchain_core.runnables import RunnableLambda, RunnablePassthrough\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "# Initialize the LLM with the desired configuration.\n",
    "llm = ChatOpenAI(model_name=\"gpt-4o\", temperature=0)\n",
    "\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Tool Definitions\n",
    "# -----------------------------------------------------------\n",
    "def resource_recommender(input_data: Dict[str, Any]) -> Dict[str, Any]:\n",
    "    \"\"\"\n",
    "    Recommend learning resources based on the user's current and target skills\n",
    "    for an AI engineering career.\n",
    "\n",
    "    Expected Input:\n",
    "      {\n",
    "          \"current_skills\": List[str],\n",
    "          \"target_skills\": List[str]  // Default: [\"Python\", \"Linear Algebra\", \"Deep Learning\"]\n",
    "      }\n",
    "\n",
    "    Returns:\n",
    "      {\n",
    "          \"recommendations\": List[{\n",
    "              \"skill\": str,\n",
    "              \"resource\": str,\n",
    "              \"url\": str\n",
    "          }]\n",
    "      }\n",
    "    \"\"\"\n",
    "    current_skills = input_data.get(\"current_skills\", [])\n",
    "    target_skills = input_data.get(\n",
    "        \"target_skills\", [\"Python\", \"Linear Algebra\", \"Deep Learning\"]\n",
    "    )\n",
    "    recommendations = []\n",
    "\n",
    "    for skill in target_skills:\n",
    "        if skill not in current_skills:\n",
    "            if skill == \"Python\":\n",
    "                recommendations.append(\n",
    "                    {\n",
    "                        \"skill\": skill,\n",
    "                        \"resource\": \"Python for Everybody (Coursera)\",\n",
    "                        \"url\": \"https://www.coursera.org/specializations/python\",\n",
    "                    }\n",
    "                )\n",
    "            elif skill == \"Linear Algebra\":\n",
    "                recommendations.append(\n",
    "                    {\n",
    "                        \"skill\": skill,\n",
    "                        \"resource\": \"Khan Academy Linear Algebra\",\n",
    "                        \"url\": \"https://www.khanacademy.org/math/linear-algebra\",\n",
    "                    }\n",
    "                )\n",
    "            elif skill == \"Deep Learning\":\n",
    "                recommendations.append(\n",
    "                    {\n",
    "                        \"skill\": skill,\n",
    "                        \"resource\": \"Deep Learning Specialization (Coursera)\",\n",
    "                        \"url\": \"https://www.coursera.org/specializations/deep-learning\",\n",
    "                    }\n",
    "                )\n",
    "    return {\"recommendations\": recommendations}\n",
    "\n",
    "\n",
    "def timeline_generator(input_data: Dict[str, Any]) -> Dict[str, Any]:\n",
    "    \"\"\"\n",
    "    Generate a milestone-based study timeline using user inputs.\n",
    "\n",
    "    Expected Input:\n",
    "      {\n",
    "          \"current_skills\": List[str],\n",
    "          \"weeks_available\": int  // Default: 12\n",
    "      }\n",
    "\n",
    "    Returns:\n",
    "      {\n",
    "          \"timeline\": List[{\n",
    "              \"week\": int,\n",
    "              \"goal\": str,\n",
    "              \"duration\": str\n",
    "          }]\n",
    "      }\n",
    "    \"\"\"\n",
    "    current_skills = input_data.get(\"current_skills\", [])\n",
    "    weeks_available = input_data.get(\"weeks_available\", 12)\n",
    "    timeline = []\n",
    "    week = 1\n",
    "\n",
    "    # Schedule Python Basics if not already mastered.\n",
    "    if \"Python\" not in current_skills:\n",
    "        timeline.append({\"week\": week, \"goal\": \"Python Basics\", \"duration\": \"2 weeks\"})\n",
    "        week += 2\n",
    "\n",
    "    # Define subsequent study milestones.\n",
    "    timeline.extend(\n",
    "        [\n",
    "            {\n",
    "                \"week\": week,\n",
    "                \"goal\": \"Linear Algebra Fundamentals\",\n",
    "                \"duration\": \"2 weeks\",\n",
    "            },\n",
    "            {\n",
    "                \"week\": week + 2,\n",
    "                \"goal\": \"Deep Learning Foundations\",\n",
    "                \"duration\": \"4 weeks\",\n",
    "            },\n",
    "            {\"week\": week + 6, \"goal\": \"Practical Projects\", \"duration\": \"4 weeks\"},\n",
    "        ]\n",
    "    )\n",
    "    return {\"timeline\": timeline}\n",
    "\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Define the Unified Prompt Template\n",
    "# -----------------------------------------------------------\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \"\"\"You are a specialized agent with the following identity and operational parameters:\n",
    "\n",
    "Identity:\n",
    "- Role: AI Career Planning Specialist\n",
    "- Domain: AI Research and Engineering Education\n",
    "- Purpose: Create personalized AI career preparation plans for users.\n",
    "- Interaction Style: Structured, supportive, and detail-oriented.\n",
    "\n",
    "Behavioral Guidelines:\n",
    "- Primary Goals:\n",
    "  - Analyze the user's current skill gaps in AI/ML.\n",
    "  - Recommend targeted and accessible learning resources.\n",
    "  - Generate a realistic study timeline with clear milestones.\n",
    "- Constraints:\n",
    "  - Use only verified and accessible resources.\n",
    "  - Consider available study time and user’s current skill level.\n",
    "- Success Criteria:\n",
    "  - Provide a clear progression path with measurable milestones.\n",
    "  - Deliver recommendations that align with the user's profile.\n",
    "\n",
    "Working Memory Management:\n",
    "- Current Task Context:\n",
    "  - Active objective: \"Create a personalized AI career study plan.\"\n",
    "  - Task progress: \"Analyzing user input and preparing recommendations.\"\n",
    "  - Pending actions: \"Match resources and generate timeline.\"\n",
    "- Recent Interaction:\n",
    "  - Last user input: {input}\n",
    "  - Conversation history: {history}\n",
    "\n",
    "Tool Integration:\n",
    "- Resource Recommender: To identify learning resources based on skill gaps.\n",
    "- Timeline Generator: To build a structured study schedule.\n",
    "\n",
    "Output Requirements:\n",
    "- Response must be in Markdown with the following sections:\n",
    "  1. Skill Gap Analysis\n",
    "  2. Recommended Resources (with URLs)\n",
    "  3. Study Timeline with milestones\n",
    "  4. Next steps and progress tracking\n",
    "\n",
    "Current Date: February 12, 2025\n",
    "Relevant Information:\n",
    "{history}\n",
    "\"\"\",\n",
    "        ),\n",
    "        MessagesPlaceholder(variable_name=\"history\"),\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Memory Initialization with Robust Error Handling\n",
    "# -----------------------------------------------------------\n",
    "try:\n",
    "    memory = ConversationKGMemory(llm=llm)\n",
    "except Exception as e:\n",
    "    print(f\"Memory initialization error: {e}\")\n",
    "    memory = None\n",
    "\n",
    "\n",
    "def load_memory(input_dict: Dict[str, Any]) -> List[Any]:\n",
    "    \"\"\"\n",
    "    Load conversation history from memory ensuring it is returned as a list.\n",
    "    Returns an empty list if the memory is unavailable or parsing fails.\n",
    "    \"\"\"\n",
    "    if memory is None:\n",
    "        return []\n",
    "    try:\n",
    "        history = memory.load_memory_variables(\n",
    "            {\"input\": input_dict.get(\"input\", \"\")}\n",
    "        ).get(\"history\", [])\n",
    "        return history if isinstance(history, list) else []\n",
    "    except Exception as e:\n",
    "        print(f\"Memory loading error: {e}\")\n",
    "        return []\n",
    "\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Response Formatter Function\n",
    "# -----------------------------------------------------------\n",
    "def format_response(llm_response: Any) -> str:\n",
    "    \"\"\"\n",
    "    Format the output from the language model or tool chain into a structured Markdown response.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        if hasattr(llm_response, \"content\"):\n",
    "            return llm_response.content\n",
    "        if isinstance(llm_response, str):\n",
    "            return llm_response\n",
    "        if isinstance(llm_response, dict):\n",
    "            recommendations = llm_response.get(\"recommendations\", [])\n",
    "            timeline = llm_response.get(\"timeline\", [])\n",
    "\n",
    "            md_response = \"# AI Engineering Career Study Plan\\n\\n\"\n",
    "            md_response += \"## Recommended Resources\\n\"\n",
    "            for rec in recommendations:\n",
    "                md_response += (\n",
    "                    f\"- **{rec['skill']}**: [{rec['resource']}]({rec['url']})\\n\"\n",
    "                )\n",
    "\n",
    "            md_response += \"\\n## Study Timeline\\n\"\n",
    "            for item in timeline:\n",
    "                md_response += (\n",
    "                    f\"- Week {item['week']}: {item['goal']} ({item['duration']})\\n\"\n",
    "                )\n",
    "            return md_response\n",
    "        return str(llm_response)\n",
    "    except Exception as e:\n",
    "        return f\"Error formatting response: {str(e)}\\nRaw response: {str(llm_response)}\"\n",
    "\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Build the Runnable Chain\n",
    "# -----------------------------------------------------------\n",
    "chain = (\n",
    "    {\n",
    "        \"input\": RunnablePassthrough(),  # Directly pass the user input.\n",
    "        \"history\": RunnableLambda(load_memory),  # Load conversation history.\n",
    "        \"recommendations\": RunnableLambda(\n",
    "            lambda x: resource_recommender(x)\n",
    "        ),  # Execute resource recommender tool.\n",
    "        \"timeline\": RunnableLambda(\n",
    "            lambda x: timeline_generator(x)\n",
    "        ),  # Execute timeline generator tool.\n",
    "    }\n",
    "    | prompt  # Feed consolidated input, history, and tool outputs to the prompt.\n",
    "    | llm  # Generate the refined response using the LLM.\n",
    "    | RunnableLambda(format_response)  # Format the final output into Markdown.\n",
    ")\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Example Usage: Generate a Study Plan for AI Research/Engineering Career Preparation\n",
    "# -----------------------------------------------------------\n",
    "if __name__ == \"__main__\":\n",
    "    try:\n",
    "        # Example user input with dynamic parameters indicated by placeholders.\n",
    "        user_input = {\n",
    "            \"input\": \"I want to become an AI engineer. I know {current_skills}.\",\n",
    "            \"current_skills\": [\"Basic Python\"],\n",
    "            \"target_skills\": [\"Python\", \"Linear Algebra\", \"Deep Learning\"],\n",
    "            \"weeks_available\": 12,\n",
    "        }\n",
    "\n",
    "        # Execute the chain and print the formatted response.\n",
    "        response = chain.invoke(user_input)\n",
    "        print(response)\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"Error executing chain: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PROMPT_QUALITY_EVALUATION_TEMPLATE\n",
    "\n",
    "This template is designed to evaluate the quality of generated prompts produced for AI agent tasks. It follows a two-part structure:\n",
    "\n",
    "1. **Prompt Generation:** Creates a high-quality prompt based on a user-defined goal.\n",
    "2. **Prompt Evaluation:** Uses an evaluation template to review the generated prompt against key criteria and returns structured feedback.\n",
    "\n",
    "A key note for this template is that every literal ```{``` and ```}``` inside the JSON block has been replaced with ```{{``` and ```}}``` to ensure that Python’s ```.format()``` method does not mistakenly treat them as placeholders.\n",
    "\n",
    "---\n",
    "\n",
    "### Part 1: Prompt Generation\n",
    "\n",
    "- **Objective:**  \n",
    "  A prompt is generated for the purpose of creating a study plan for becoming an AI engineer. This generated prompt is meant to guide an AI agent in creating a comprehensive career preparation plan.\n",
    "\n",
    "- **Implementation Details:**\n",
    "  - A **generation template** is created using ```ChatPromptTemplate.from_messages()```.  \n",
    "  - The system message instructs the agent (acting as a Prompt Generator) on the required components of the prompt:\n",
    "    - Agent Identity and Role\n",
    "    - Behavioral Guidelines and Success Criteria\n",
    "    - Task Description and Input Parameters\n",
    "    - Instructions for memory management and tool integration\n",
    "  - A human message accepts the ```{user_goal}``` as input.\n",
    "  - The ```user_goal_text``` variable contains an example goal where the user specifies they want to develop a study plan targeting improvements in Python programming, data analysis, and fundamental machine learning concepts.\n",
    "  - The prompt is then generated by calling ```generation_template.format_messages(user_goal=user_goal_text)``` and concatenated into a single string (```generated_prompt_str```).\n",
    "\n",
    "---\n",
    "\n",
    "### Part 2: Prompt Evaluation\n",
    "\n",
    "- **Objective:**  \n",
    "  The evaluation template reviews the generated prompt to ensure that it adheres to best practices for prompt quality. It checks if various aspects of the prompt are well defined and actionable.\n",
    "\n",
    "- **Implementation Details:**\n",
    "  - An **evaluation template** is defined using ```ChatPromptTemplate.from_messages()``` with a system message that:\n",
    "    - Sets the task by defining the role of a **Prompt Quality Reviewer**.\n",
    "    - Provides the **Content to Review** using a ```{generated_prompt}``` placeholder.\n",
    "    - Outlines **Verification Steps** that cover:\n",
    "      1. **Identity and Role Clarity:** Checks if the agent’s role, purpose, and behavioral guidelines are clear.\n",
    "      2. **Memory Management:** Assesses the structure for managing context.\n",
    "      3. **Tool Integration:** Reviews the mention and use of necessary tools.\n",
    "      4. **Plan Completeness:** Ensures all aspects required for the task are present, including clear input parameters.\n",
    "      5. **Quality Standards:** Evaluates clarity, conciseness, and overall effectiveness.\n",
    "    - Specifies a **Return Format** in JSON. Notice the use of double curly braces (```{{``` and ```}}```) in the JSON block. This prevents Python’s string formatter from interpreting them as variable placeholders. It outlines expected keys like ```\"score\"```, ```\"feedback\"``` (with sub-fields like ```identity_clarity```, ```memory_management```, etc.), and an array of ```\"improvement_suggestions\"```.\n",
    "    - Provides an **Example Response** within the template to illustrate how the evaluation should be formatted.\n",
    "  - A human message passes the ```{generated_prompt}``` into the evaluation template.\n",
    "  - The code then calls ```evaluation_template.format_messages(generated_prompt=generated_prompt_str)``` to properly inject the generated prompt into the evaluation template.\n",
    "  - Finally, the evaluation prompt is sent to the LLM (```ChatOpenAI```) that returns a structured evaluation response based on the defined criteria.\n",
    "\n",
    "---\n",
    "\n",
    "### How It Works Together\n",
    "\n",
    "1. **Generation:**  \n",
    "   The user goal is processed to create a detailed prompt that instructs an AI agent on how to build a personalized study plan. The strategy is to ensure that all necessary details (like identity, behavioral guidelines, and instructions for memory and tool integration) are clearly defined.\n",
    "\n",
    "2. **Evaluation:**  \n",
    "   Once generated, the prompt is reviewed by another sub-agent (acting as a Prompt Quality Reviewer). The evaluation checks for clarity, completeness, and technical feasibility based on the provided criteria.\n",
    "\n",
    "3. **JSON Formatting:**  \n",
    "   To avoid conflict with Python’s string formatting, all literal curly braces in the JSON block are doubled. This ensures that the evaluation response is formatted as intended.\n",
    "\n",
    "---\n",
    "\n",
    "This practical example demonstrates a modular approach where one agent generates a prompt while another evaluates its quality. Both parts integrate with LangChain's templating system, allowing for the creation of dynamic, high-quality prompts that can adapt based on user input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated Prompt:\n",
      " You are a Prompt Generator specialized in crafting effective prompts for AI agents.\n",
      "Your task is to create a clear and structured prompt based on the user's goal.\n",
      "Ensure the generated prompt includes the following elements:\n",
      "- Agent Identity and Role\n",
      "- Behavioral Guidelines and Success Criteria\n",
      "- Task Description and Input Parameters\n",
      "- Any necessary instructions for memory management and tool integration\n",
      "\n",
      "I want to develop a study plan for becoming an AI engineer. My focus areas are improving my Python programming, learning data analysis, and understanding machine learning fundamentals. Please generate a prompt that guides an AI agent to create a comprehensive career preparation plan.\n",
      "\n",
      "Evaluation Response:\n",
      " {\n",
      "  \"score\": 78,\n",
      "  \"feedback\": {\n",
      "    \"identity_clarity\": \"The agent's role as a career preparation planner is clear, but the behavioral guidelines could be more detailed.\",\n",
      "    \"memory_management\": \"The prompt does not explicitly address memory management, which could lead to context loss in complex tasks.\",\n",
      "    \"tool_integration\": \"There is no mention of specific tools or resources that the agent should use or recommend.\",\n",
      "    \"plan_completeness\": \"The prompt covers the main focus areas but lacks detailed guidance on how to structure the study plan.\",\n",
      "    \"quality_standards\": \"The prompt is generally clear but could benefit from more concise language and specific instructions.\"\n",
      "  },\n",
      "  \"improvement_suggestions\": [\n",
      "    \"Define specific behavioral guidelines for the agent, such as prioritizing user preferences and providing actionable steps.\",\n",
      "    \"Include instructions for managing memory, such as summarizing previous interactions or storing key information.\",\n",
      "    \"Suggest specific tools or resources, like online courses or programming environments, to enhance the study plan.\"\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "# Initialize the LLM with the desired configuration.\n",
    "llm = ChatOpenAI(model_name=\"gpt-4o\", temperature=0)\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Part 1: Prompt Generation\n",
    "# -----------------------------------------------------------\n",
    "# Define the generation prompt template with a system message and a human message.\n",
    "generation_template = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \"\"\"You are a Prompt Generator specialized in crafting effective prompts for AI agents.\n",
    "Your task is to create a clear and structured prompt based on the user's goal.\n",
    "Ensure the generated prompt includes the following elements:\n",
    "- Agent Identity and Role\n",
    "- Behavioral Guidelines and Success Criteria\n",
    "- Task Description and Input Parameters\n",
    "- Any necessary instructions for memory management and tool integration\n",
    "\"\"\",\n",
    "        ),\n",
    "        (\"human\", \"{user_goal}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "# Example user goal describing the purpose.\n",
    "user_goal_text = (\n",
    "    \"I want to develop a study plan for becoming an AI engineer. \"\n",
    "    \"My focus areas are improving my Python programming, learning data analysis, \"\n",
    "    \"and understanding machine learning fundamentals. \"\n",
    "    \"Please generate a prompt that guides an AI agent to create a comprehensive career preparation plan.\"\n",
    ")\n",
    "\n",
    "# Generate the prompt from the template.\n",
    "generated_prompt_messages = generation_template.format_messages(\n",
    "    user_goal=user_goal_text\n",
    ")\n",
    "# Join the message contents into a single string (avoiding the 'list' object error).\n",
    "generated_prompt_str = \"\\n\".join(msg.content for msg in generated_prompt_messages)\n",
    "print(\"Generated Prompt:\\n\", generated_prompt_str)\n",
    "\n",
    "# -----------------------------------------------------------\n",
    "# Part 2: Prompt Evaluation\n",
    "# -----------------------------------------------------------\n",
    "# Define the evaluation template.\n",
    "evaluation_template = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \"\"\"### Task\n",
    "You are a Prompt Quality Reviewer tasked with verifying the quality of agent prompts.\n",
    "\n",
    "### Content to Review\n",
    "{generated_prompt}\n",
    "\n",
    "### Verification Steps\n",
    "1. Identity and Role Clarity  \n",
    "   - Is the agent's role and purpose clearly defined?  \n",
    "   - Are behavioral guidelines specific and actionable?\n",
    "\n",
    "2. Memory Management  \n",
    "   - Is the prompt structured to track context and manage memory effectively?\n",
    "\n",
    "3. Tool Integration  \n",
    "   - Are any necessary tools and their usages mentioned clearly?\n",
    "\n",
    "4. Plan Completeness  \n",
    "   - Does the prompt cover all essential aspects required for the intended task?  \n",
    "   - Is there clear guidance on expected input parameters?\n",
    "\n",
    "5. Quality Standards  \n",
    "   - Is the prompt clear, concise, and effective?\n",
    "\n",
    "### Return Format\n",
    "Provide your evaluation in JSON format as follows:\n",
    "\n",
    "```\n",
    "{{\n",
    "  \"score\": <integer_between_0_and_100>,\n",
    "  \"feedback\": {{\n",
    "    \"identity_clarity\": \"<feedback>\",\n",
    "    \"memory_management\": \"<feedback>\",\n",
    "    \"tool_integration\": \"<feedback>\",\n",
    "    \"plan_completeness\": \"<feedback>\",\n",
    "    \"quality_standards\": \"<feedback>\"\n",
    "  }},\n",
    "  \"improvement_suggestions\": [\n",
    "    \"<suggestion_1>\",\n",
    "    \"<suggestion_2>\",\n",
    "    \"<suggestion_3>\"\n",
    "  ]\n",
    "}}\n",
    "```\n",
    "\n",
    "### Example Response\n",
    "```\n",
    "{{\n",
    "  \"score\": 85,\n",
    "  \"feedback\": {{\n",
    "    \"identity_clarity\": \"Agent role and behavioral guidelines are well defined, but success criteria could be more specific.\",\n",
    "    \"memory_management\": \"The working memory structure is comprehensive but lacks clear cleanup protocols.\",\n",
    "    \"tool_integration\": \"Tool selection is robust; however, error handling details could be improved.\",\n",
    "    \"plan_completeness\": \"All essential elements are present, but additional details on input parameters are needed.\",\n",
    "    \"quality_standards\": \"The prompt is generally clear, though conciseness can be improved.\"\n",
    "  }},\n",
    "  \"improvement_suggestions\": [\n",
    "    \"Add specific memory cleanup triggers.\",\n",
    "    \"Include detailed error recovery procedures.\",\n",
    "    \"Specify maximum retry attempts for failed operations.\"\n",
    "  ]\n",
    "}}\n",
    "```\n",
    "\"\"\",\n",
    "        ),\n",
    "        (\"human\", \"{generated_prompt}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "# Format the evaluation prompt with the generated prompt inserted.\n",
    "evaluation_messages = evaluation_template.format_messages(\n",
    "    generated_prompt=generated_prompt_str\n",
    ")\n",
    "\n",
    "# Invoke the LLM to evaluate the generated prompt.\n",
    "evaluation_response = llm.invoke(evaluation_messages)\n",
    "print(\"\\nEvaluation Response:\\n\", evaluation_response.content)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain-opentutorial-A2cWC-y3-py3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
